Web Robot Detection in Academic Publishing

Recent industry reports assure the rise of web robots which comprise more than half of the total web trac. Œey not only threaten the security, privacy and eciency of the web but they also distort analytics and metrics, doubting the veracity of the information being promoted. In the academic publishing domain, this can cause articles to be faulty presented as prominent and inƒuential. In this paper, we present our approach on detecting web robots in academic publishing websites. We use di‚erent supervised learning algorithms with a variety of characteristics deriving from both the log €les of the server and the content served by the website. Our approach relies on the assumption that human users will be interested in speci€c domains or articles, while web robots crawl a web library incoherently. We experiment with features adopted in previous studies with the addition of novel semantic characteristics which derive a‰er performing a semantic analysis using the Latent Dirichlet Allocation (LDA) algorithm. Our real-world case study shows promising results, pinpointing the signi€cance of semantic features in the web robot detection problem.